Information theory, 
pattern recognition, 
and neural networks 


Lecture 1: 
Introduction to 
Information Theory 
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Q: A file of N = 10000 bits is stored on 
this disc drive (with f = 0.1), then read. 


Roughly how many bits are flipped? 
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Q: то make a successful business selling 1 Gigabyte disc drives, 
how small does the flip probability f need to be? 
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Repetition code 'R3' 
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Repetition code 'R3' 
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Repetition code 'R3' 
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Performance of repetition codes 
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What's achievable? 


Block Cen 4, 


Shannon's noisy-channel coding theorem 
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Course outline 
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Information Theory, inference, 
and Learning Algorithms 


www.inference. 


Information Theory 
‘Source coding (Data compression) 


з Noisy-channel coding 


Data modelling, pattern recognition, 


learning & memory 


“Tools 
» Exact inference methods 
» Monte Carlo methods 
» Variational methods 


Neural networks 
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The weighing problem 


ОО 
ОО 


do 
You are given 12 balls, all equal in weight except for one 
that is either heavier or lighter. 
Design a strategy to determine 
which is the odd ball 
and whether it is heavier or lighter, 
in as few uses of the balance as possible. 


